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ABSTRACT 



Recent years have seen many attempts to program learning 
based on many principles ranging from intuition to mathematics. 
The area of criterion establishment for advancement or con- 
versely additional training at the same or lower levels remains 
primarily in the realm of intuition. The intuition used is 
only as good as the experience and background of the program 
designer however. In the case of a very experienced program 
designer the criteria may be very efficient. In the case of 
less experienced program designers the criteria are usually 
arbitrary. This work describes a method of analyzing learning 
programs and determining mathematically sound criteria. The 
mathematical foundation for this analysis is the Markovian 
learning model as opposed to the linear learning model. 
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I. 



INTRODUCTION 



A. GENERAL 

This work was stimulated through the author's association 
with the Behavioral Sciences Institute of Monterey, California. 
Members of the staff of the Institute have developed a pro- 
cedure for preparing language training programs [Gray and 
Ryan 1973] . This procedure "Language Training Through 
Programmed Conditioning" is based on a mathematical principle 
or structure. The total language program consists of ninety 
sub-programs covering different language deficiencies which 
are the result of various causes such as deafness, mental 
retardation, non-English speaking, etc. Each sub-program 
consists of a series of steps that sequentially train a 
student until he reaches a desired objective. The total pro- 
gram is designed to test the student in a continuous manner 
to determine what sub-programs or parts (steps) he needs, 
when he has completed any step or complete sub-program, and 
when he is having excessive problems that require stepping 
back (branching) or other forms of special help. 

The procedure, though developed for language training, 
probably has wide application. These applications will appear 
as more people, in varied fields, become familiar with the 
procedure and investigate its appropriateness to their specific 
areas of interest. Along this vein it is also hoped that the 
method developed herein, of analyzing the program and 
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determining criteria to be used in the administration of the 
program, will have wider application. 

B. PROGRAMMING 

The objective of an intellectually oriented training 
program is to move a subject from a specific unlearned state 
(U) to a learned state (L) . The process used to achieve this 
goal is Stimulus-Response-Consequence, normally referred to 
as operant conditioning. In most situations the desired 
learned state (L) can be achieved most effectively by reducing 
the unlearned state (U) to a number of small steps (states) 
which are learned in a sequence and build up to the ultimate 
goal. This latter process is called programming. Throughout 
this work the term presolution will be used interchangeably 
with the unlearned state and the term solution with the 
learned state. 

C. CRITERIA 

The subject of criteria arises when the programmer desires 
to measure the success of a given program in moving a specific 
subject from the unlearned to the learned state. Basically the 
question is: "What measure should be used for determining 

when a subject has achieved the learned state (L) ?" A second 
aspect of criteria arises when it appears that a subject is 
having an excessive amount of trouble with a particular step. 

In such a case it may be desirable to alter the program for 
the subject, possibly reducing the step to several less dif- 
ficult substeps. The first criterion is referred to as 
advancement criterion and the second is referred to as 
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branching criterion. A difficulty in establishing these 
criteria results from the generally accepted idea that there 
is no exact (deterministic) method of establishing criteria. 
Alternately learning is viewed as being probabilistic (sto- 
chastic) in nature and therefore the criteria must be 
established within the realm of probability theory. To 
further complicate the situation there is not a universally 
accepted learning theory and therefore the criteria, if 
determined statistically, will be a function of the learning 
theory the programmer subscribes to. 

Some possible measures for criteria are: total number of 

correct responses, total number of incorrect responses, total 
number of trials, plus the associated ratios and percentages. 

A trial is one Stimulus-Response-Consequence sequence. Of 
these it seems intuitively appealing to select total number 
or percentage of correct responses as an indication of when 
to advance a subject. More specifically it is common practice 
to use a string of consecutive correct responses for advance- 
ment. Determining the number in this string and the confidence 
in that number is required in establishing the first criterion. 
Determining a maximum acceptable number of trials and/or 
errors per' step, and the associated confidences, is the pro- 
cedure used to establish the second criterion. 



8 



II. MODEL DEVELOPMENT 



A. BACKGROUND 

In a brief historical review of mathematical learning 
theory, Atkinson, Bower, and Crothers (1965, pp. 19-24), 
indicate that the treatment of learning using probabilistic 
models began with Thurstone in 1919. From 1919 through 
1950 there were numerous probabilistic models proposed and 
tested, all related to various specific learning situations. 
Since 1950 there has been a great deal of work done in the 
area of stochastic learning, and two theories have evolved 
as the primary contenders for acceptance: the linear model 

and the Markov model. Basically the linear model states that 
a subject has a probability of success on any trial n which 
is given by the following equation: 

P n = 1 - (1-P 1 ) (1-0) n_1 (1) 

where P^ is his initial probability of success and 0 is his 
learning rate. The Markov model takes a quite different 
approach. Basically the theory says that if a subject is in 
an unlearned state (U) then the probability of a correct 
response is g (guess) , and if the subject is in the learned 
state (L) , then the probability of a correct response is 1. 

The probability of going from the unlearned state to a learned 
state on any presolution trial is usually denoted c. The 
probability of a correct response on any trial n is usu' ’.ly 
given by: 
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P n = l - (l-g) (l-c) n 1 (2) 

Comparison of Eqs . (1) and (2) indicates that their forms 

are exactly the same. The difference in these two equations 
lies in their theoretical background and the meaning of the 
parameters. Equation (1) states that a subject starts with 
a probability of making a correct response on the first 
trial. The probability of success on the second trial is 
greater, due to the incremental learning achieved on the first 
trial. This linear process continues indefinitely and the 
subject's probability of success asymptotically approaches 1. 
Equation (2) states that on each presolution trial a subject 
has a probability c of going into solution. Once in solution 
the subject will remain in solution and always respond cor- 
rectly (P n =l) . While in presolution the subject has a proba- 
bility. g (guess) of responding correctly and this probability 
does not change. The form of these two equations and other 
interesting comparisons are made in "Learning: Gradual or 

All-or-None , " [Restle and Greeno 1970, Chap. 2). Based on 
their analysis Restle and Greeno stated, "... the all-or- 
none theory is the most interesting, and we think it is the 
one most deserving of future work," (p. 79). 

Pilot research involving a computer simulation of the 
linear model yielded a set of criteria for use in the language 
training programs of the Institute. Informal observation of 
the data on which the linear model was based suggested that 
the linear model might not be appropriate. Further 
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study 1 of data from students trained on these and other 
programs showed a definite tendency toward the Markov 
principles of stationarity and independence [Atkinson, and 
others, 1965, pp. 39-49; Coombs, Dawes, and Tversky 1970, 
pp. 296-297] . Based on these results the linear model was 
set aside and this work started based on the Markovian (All- 
or-None) principle. 

B. ASSUMPTIONS 

The following assumptions are necessary to the development: 

1. The learning process involved in the programs devel- 
oped by the Institute are Markovian in nature. 

2. The subject can be correct on the first trial of 

any step by either: (1) being in solution prior to the trial, 

(2) going into solution because of the information presented 

in the first stimulus, or (3) guessing correctly in presolution. 
This assumption modifies Eq. (2) in that Eq. (2) contains the 
restriction that for the subject to be correct on the first 
response, he must guess correctly, therefore not allowing the 
possibility of being in solution (the learned state) on the 
first trial. Allowing for the possibility that the subject is 
in solution on the first trial [Atkinson, and others, 1965, 
footnote p. 55] , appears to be a more realistic approach and 
was used in this work. 



Gray, B. B.; Carrier, J. K., Jr.; Bradfield, R. H.; and 
Rigg , L . , The Discrete Effects of Response Contingent Reinforce - 
ment During Skill Acquisitio n, submitted to Journal of Applied 
Behavior Analysis for publication . 
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3. The g factor (guess) in presolution is a function of 
the conditioning sequence (step) and the subject. 

4. The c factor (probability of going into solution) is a 
function of the step and the subject. 

5. g and c are constant over any step for a given subject. 

6. The set of outcomes form a homogeneous Markov chain. 



L , . U , . P (Correct I Row State) 

n+l n+± r 



1 


0 




V 


c 


(1-c) 




_g 



n = 0,1,2, . . . 
g < 1 , c < 1 



I 



(3) 



C . MODEL 

The equations developed in this work are a combination 
of .what is presented in Refs. 1-3 and some original work. 
Since there is so much overlap it is not possible to give 
credit to any one source, instead the equations will be given 
with explanations and generally acknowledging the credit to 
all. The first development is the probability of errorless 
response given that the subject is known to be in the un- 
learned state (U) . This state is assumed on the first trial 
and known to exist if an error occurs. If a subject is put 
on a program step and no error occurs before reaching the 
advancement criterion, then there is no way of determining 
whether he was in solution (learned state, L) initially, or 
was in presolution (unlearned state, U) initially and performed 
as follows: 
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( 4 ) 



2 2 

P[Errorless] = c + g(l-c)c + g (1-c) c + ... 

P [Errorless] = -< ^ r 

n->°° l-g(l-c) 

For future reference, this probability will be called p (rho) , 
the probability of errorless response given that the subject 
is in presolution. (Being in presolution is demonstrated any- 
time a subject makes an error.) The above development says 
that the subject either goes into solution on the first trial; 
stays in presolution, guesses correctly and then goes into 
solution; stays in presolution twice, guesses correctly twice 
and then goes into solution; etc. The development indicates 
that the subject eventually goes into solution if errorless 
response is achieved after an error. The reader familiar with 
the Markov theory will note that -the term relating to remain- 
ing in presolution and having errorless response was omitted 
in developing Eq. (4) . The omission was committed since the 
term g n (l-c) n goes to zero in the limit as n approaches infinity. 

The next development will be the expected number of errors 
given g and c. The development relies on the parameter p. The 
probability that the total number of errors is exactly k 
equals : 

P[E = k] = l ( k | i )g i (l-c) i (l-g) k (l-c) k c 
i=0 • 

If this abbreviated mathematical statement were written out, it 
would represent every feasible combination of events in which 
exactly k errors can occur. 
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\ P[E = k] = 



(l-g) k (l-c) k c l ( k ^ i )g i (l-c) i 
i=0 



Referring to a standard math table e.g. CRC [13th Edition, p. 
463] . This equation then reduces to 



P [E = k] = 



Referring to Eq. (4) 



k 

[ (1-g) (1-c) ] c 

r (1-g) (1-c) ! k 
1 l-g(l-c) J 

[i - ° , k 

1-g (1-c) ^ 

and substituting 



1 k+1 

'l-gd-c) 1 

■l-gd-'cV 1 



P[E = k] = (1-P) k p (5) 

In words Eq. (5) says that there were k possible errorless 
response strings broken by the occurrence of an error and 
then there were no more errors. Since the probability of an 
errorless response string is p, given that the subject is in 
presolution, it follows that not errorless response is equal 
to (1-p) . This takes into account all possible numbers of 
correct responses before the error response that breaks the 
string. The occurrence of an error demonstrates presolution 
and also allows another possible string of errorless response, 
which is independent of the length or number of previous 
strings and is only dependent on the knowledge of being in 
presolution, i.e., the occurrence of an error. In summary 
then Eq. (5) says that the probability of exactly k errors 
occurring is equal to not errorless response k times and then 
errorless response. 



14 



E [E] 



CO 



E [E ] = 

The next development is the expected trial number of the 
last error. The probability that the last error occurred on 
trial t equals: 

P[T=0] = p 

P[T=t] = (l-c) t (l-g)p ' (8) 

t = 1,2,3, •• • 

In words, Eq. (8) says that there were t presolution trials 
indicated by an error on trial t and then errorless response. 
This probability statement allows for any sequence (order) or 
number of correct and incorrect responses up to trial t. The 
only required knowledge is that an error occurs on trial t and 
then no more errors. It should be noted that the probabilities 
given by Eqs. (5) and (8) are both decreasing functions. This 
corresponds with what will be observed about Figs. 2 and 1 
respectively. 



I k • P [E=k] = l k • p (1-p) 

k=0 k=0 



p(l-p) l k(l-p) 
• k=0 



k-1 



p(l-p) _ (1-p) 

p 2 ' p 

(l-g) (1-c) 
c 



( 6 ) 

(7) 
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oo 



E[T] ■ = 



l t • P [T=t] 
t=0 



0.p+ l t • P [T=t ] 
t=l 



l t • (1-c) t (1-g) p 
t=l 



E[T] = p (1-g) (1-c) l f (1-c) 

t=l 



t-1 



p ( l-g) (l-c) 

2 

C 

“*E [E] 



E [E] 

1-g (1-c) 



(9) 



E [T] 



(1-g) (1-c) 
[1-g (1-c) ] c 



(10) 



With Eqs. (6) and (9) it only requires a little manipulation 
to get estimates of g and c: 



from Eq. (6) 

p = (E [E] +1) 



from Eq. (9) 
c 



• • c 



P 



. E[E] 

E[T] 



E[E] 

E[T] (E[E]+1) 



• A 

• • C 



E 

T (E+l) 



(ID 



In Eq. (11) the expected number of errors E (E) is replaced 
by the observed number of errors and similarly the trial of the 
last error T replaces E(T) . This might seem out of ord to 
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some, in that a random variable is substituted for its expected 
value (mean or first moment) . Some thought, however, will 
point out that the observed values of E and T are the mean 
when the sample size is one. Each subject/step combination 
represents a unique data point that cannot be repeated since 
the completion signifies the transition from the unlearned 
state (presolution) to the learned state (solution) , and g and 
c factors are no longer applicable when the subject is in solu- 
tion. This procedure represents a special case of the applica- 
tion of method of moments in estimating the parameters of a 
distribution: 



from Eq. (9) g 



E[E] 

eTtT 



Substituting E=E(E) and T=E(T) and utilizing Eq. (11): 



T - E 
T 



g 




(T-E) (E+l) 
T (E+l) -E 



• ^ 
• • g 



T (E+l) -E 



( 12 ) 



If E is much greater than 1 (E>>1) then 



c 



1 

T 



Also 



1 



E 



(T-l) + § 



Since T is greater than or equal to E(T > E) 
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These equations state that c is approximated by the reciprocal 
of the number of the trial of the last error. This is intu- 
itively appealing as it states the larger the c factor 
(probability of going into solution) , the fewer expected number 
of trials. The estimate of g is slightly larger than the 
apparent presolution accuracy. This comes from the concept 
which allows the possibility of presolution correct responses 
after the last error. If the number of these responses were 

A 

known, the estimated presolution accuracy rate (g) would 
increase above what can be estimated using only the trials to 
the last error. The required correction factor is taken into 
account in Eq. (12) . 



PREFACE — SECTION III 



During the final proofreading of this thesis it was discovered that 
the computation of g (Appendix B and Figs. 4, 7, and 8) had been done 

E 2 E 2 

using: g = 1 ~ t( e TT ) 1¥ ' instead of 9 = 1 ~ T( E +lT =E (Eq ~ 12) • 

The second equation is the valid equation as it appears in the text. 
Specific reference to g, as it applies to the data analyzed, must therefore 
be considered in light of the first equation, e.g., pages 20 and 21. 

The effect of using the first equation is to cause an across-the-board 
increase in g. The general conclusions are still valid but the spec fic 
conclusions as regards the data analyzed, must be viewed in light of this 
new .information. Examples of the effect of this change are: (1) a eneral 

shift of Figs. 4, 7, and 8 to the left, (2) a general decrease in the mean 
values of g listed in Tables I and II, (3) a general decrease in the esti- 
mated values listed in Table III. This is based on observing that the 
slope of the associated equation (page 25, no number) is positive with 
respect to g. This would move the estimated value toward the observed 
(12% estimated and 8% observed) , and (4) a general increase in the esti- 
mated values listed in Tables IV and V. This is based on observing that 
the slope of Eqs. (15) and (16) are negative with respect to g. This would 
move the estimated values toward the observed. 
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III. APPLICATION OF THE MODEL 



A. DATA SOURCE 

With the estimates of g and c being obtainable from only 
a knowledge of the total number of errors and the trial num- 
ber of the last error, it was now a matter of obtaining some 
data to evaluate the model. The Institute has 41 programs in 
operation. These programs were being used in 20 different 
locations in the country. Due to the nature of the overall 
program, individual students receive only those programs which 
they require. It was therefore difficult to determine which 
program to sample. After some consideration the "What is" 
program was decided on. Calls were sent out to four sites for 
complete data on students having completed this program. There 
were 19 subjects from the 4 locations, the least from any loca- 
tion was 2 and the most was 7. There were 11 boys and 8 girls. 

The "What is" program consists of 35 steps grouped into 9 
series. With 19 subjects there was a possibility of obtaining 
(35x19) = 765 sets of data points, each representing a unique 
student/step combination. Throughout the remainder of this 
work "data point" will refer to a set of data from a unique 
student/step combination. Only 208 data points were available 
for the following reasons: 

1. Due to the placement procedure the student may 
start at a point other than the beginning of the program. 

(There were 208 missing points due to this.) 



19 



2. If a student is put through a step and makes no 
errors, there is no way to tell whether he was in solution 
before he started or had an errorless response string. 

Therefore, these points yielded no usable information and had 
to be ignored. (There were 349 of these points.) 

B. EXAMINATION OF DATA 

The raw data formed a 35 by 19 matrix with 208 data points. 
Each data point consisted of the trial number of the last error 
(T) and the total number of errors (E) . The data was then 

A A 

converted into a similar matrix containing c and g at each 
data point using Eqs. (11) and (12) . The first step was to 
examine the various frequency distributions to see what informa- 
tion they would yield. Frequency distributions are presented for 
the number of trials (T) , number of errors (E) , c and g. These 
distributions consisted of a total of 208 data points each and 
are shown in Figs. 1, 2, 3, and 4 respectively. Figure 1 shows 
the frequency distribution of the trial number of the last 
error, the interval widths are two trials. Figure 2 shows the 
frequency distribution of the number of errors, the interval 
widths are one error. Figures 3 and 4 show the frequency 

A A 

distribution of c and g respectively, the interval widths are 
.04. Figures 1, 2, and 3 exhibit a consistent decreasing 
nature, which indicates the lower values of T, E, and c are 
predominant in the sample and should generally be expected if 
the sample is representative of the population. Figure 4 has 
a much different shape, somewhat similar to a truncated normal 
type distribution. Figures 3 and 4 each have one point hat 
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breaks the otherwise consistent pattern. These points are 

A A 

in the interval .48 to .52 for c, and .64 to .68 for g. There 
appears to be a relatively simple explanation for these points: 
Data points having 1 error and 1 trial in presolution, that is 

A ^ 

an error on the first trial only, yields a c = .50, and a g = 
.667. If a student was in solution, i.e., knew how to 
answer correctly, but misunderstood the protocol the chances 
are good that he would have one error only. This protocol 
concept seems a valid explanation for the cut of place peaks 
in Figs. 3 and 4. 

Examination of the individual data (Appendixes A and B) 

A A 

showed no consistent trends. All of the factors T, E, c and g 
vary both over subjects and over steps. Accordingly the next 
approach was to calculate the means and standard deviations of 

A ^ 

c and g by subject and by step. The results are shown in 
Tables I and II, respectively. The frequency distributions of 
these values are plotted in Figs. 5 through 8. Figures 5 and 

A 

6 show c by subject and step, respectively. It is apparent 
that these frequency distributions overlap and that they fall 

A 

under the overall frequency distribution of c shown in Fig. 3. 
Similarly Figs. 7 and 8 overlap and fall under the overall 
frequency distribution of g shown in Fig. 4. 

It had been hoped that some definite consistency in one of 
the parameters would show up over subjects or steps. It appears, 
however, that the parameters g and c are random variables from 
the overall distributions shown in Figs. 3 and 4. Assuming 
that the sample is representative, it then follows that n this 
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TABLE I 



MEAN AND 


STANDARD 


DEVIATION OF 


g AND c 


BY SUBJECT 






A 

g 




A 

c 


Subject 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


1 


.738 


.167 


.023 


.019 


2 


.712 


.283 


.060 


.040 


3 


.789 


.119 


.109 


.149 


4 


.752 


.143 


.081 


.065 


5 


.840 


.038 


.144 


.077 


6 


.769 


.174 


.148 


.154 


7 

8 

‘ 9 


.764 


.085 


.225 


.240 


.725 


.245 


.095 


.105 


10 


. 824 


.099 


.083 


.139 


11 


.885 


.135 


.154 


.214 


12 


.759 


.119 


.057 


.063 


13 


.767 


.144 


.025 


.023 


14 


.811 


.136 


.230 


.248 


15 


.741 


.146 


.048 


.044 


16 


.790 


.128 


.092 


.132 


17 


. 805 


.105 


.173 


.197 


18 


.837 


.101 


.119 


.191 


19 


.667 


0 


.500 


0 
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TABLE II 



MEAN AND STANDARD DEVIATION OF g AND C BY STEP 



A 

g 


A 

c 


A 

g 


A 

c 


Step 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Step 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


1 


.554 




.260 


.183 


.190 


19 


.819 


.114 


.063 


.059 


2 


. 667 




.097 


.315 


. 216 


20 


.736 


.205 


.063 


.055 


3 


.635 




.125 


.268 


.255 


21 


.797 


.086 


.067 


.087 


4 


.723 


.196 


.189 


.271 


22 


.760 


.214 


.070 


.057 


5 


.792 




.185 


.085 


.123 


23 


.794 


.066 


.020 


.020 


6 


.876 




.065 


.085 


.094 


24 


.719 


.093 


.136 


.202 


7 


.758 


.112 


.062 


.091 


25 


.764 


.147 


.092 


.120 


8 


.812 




. i .26 


.182 


.276 


26 


.775 


.081 


.041 


.056 


9 


. 852 


.085 


.051 


.077 


27 


.821 


.038 


.054 


.028 


10 


.884 




.049 


.065 


.072 


28 


.835 


.106 


.091 


.095 


11 


.809 




.141 


.036 


.027 


29 


. 882 


.075 


.043 


.038 


12 


.774 




.124 


.205 


.259 


30 


.842 


.095 


.072 


.051 


13 


.764 




.192 


.053 


.051 


31 


.777 


.122 


.136 


.185 


14 


.746 


.159 


.131 


.128 


32 


.747 


.143 


.027 


.025 


15 


.742 


.126 


. 021 


.013 


33 


. 851 


.085 


.068 


.054 


16 


.852 


.185 


.054 


.025 


34 


.855 


.111 


.093 


.100 


17 


.767 


.218 


. 040 


.032 


35 


.834 


.100 


.037 


.026 


18 


. 772 


.128 


.045 


.029 
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V 



program the values of g and c, and therefore L and E, are 
random variables from the general populations indicated by 
Figs. 1 through 4. 

C. CRITERIA DETERMINATION 

Based on this conclusion, general criteria for all students 
on this program can be determined. Two criteria have to be 
established: (1) At what point in the program can the student 

be advanced with a high probability of being in solution? 

(2) At what point in the program should a student be sent 
to a subroutine which has lower difficulty levels? 

1 . Advancement Criterion 

The first criterion is based on the student's g level 
for that step and can be analyzed from the individual data 
given in Appendix B and Fig. 4. The individual data (Appendix 

A 

B) showed that the largest g for this population is .974. Using 
this as an indication of the highest g that might be expected, 
the next step is to determine its approx, probability of occur- 

A 

rence [P(g=M)]. Figure 4 shows that there were 8 times when g 
was recorded between .96 and 1.0. This represents approximately 
.04 or 4 percent. 

P[g=.98] - .04 

Table III gives an indication of what the probability 
would be of not being in solution. The values listed are for 
several different consecutive numbers of correct responses 
(X) as the criterion. M is the midpoint value of g used in 
Fig. 4, i.e., (.98, .94, .92 ) and is used to repres nt 
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TABLE III 



ESTIMATED PROBABILITY OF NOT BEING IN SOLUTION 
[P (S) ] ONCE THE CRITERION OF X CONSECUTIVE 
CORRECT RESPONSE HAS BEEN ACHIEVED 


Criterion 


Estimate 


X 


P(S) 


8 


.26 


10 


.21 


12 


.17 


14 


i — 1 
• 


16 


.12 


18 


o 

H 

• 



the interval. The following basic, conditional probability 
statement was used for these calculations. 

P [Not Solution] =P [S’] =£p [Not Solution | g=M ,Criterion=X] P [g=M] 

M 

P[S]=£(M) X *P[g=M] 

M 

It is obvious that the criterion must be established 
based on the desired certainty. For purposes of validation, 
let the criterion for completion be established at 16 con- 
secutive correct responses. The actual raw data criterion 
levels of 10 and 20 consecutive correct responses were used 
in a rather arbitrary fashion. Of the possible 765 data 
points, there were 765 - 208 = 557 in which the subject was 
tested. Of these 557 data points, there were 306 which sed 
a criterion level of 20. Of these 306 data points, thei j 
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were only 24 in which a criterion level of 16 would not have 
been satisfactory. That is, there were 24 data points where 
16, 17, 18, or 19 consecutive correct responses were recorded 
followed by at least one more errors. This value 
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306 



.08 = 8% 



means that 8 percent of the time the criterion of 16 was in- 
adequate. The prediction was 12 percent, and therefore the 
method of setting this first criterion level seems appropriately 
conservative . 

2 . Branching Criterion 

To establish the second criterion we turn to Figs. 1 and 
2. Figure 2 tells us that 90% of the data points had less than 
25 errors. Figure 1 tells us that 87% of the data points were 
completed in less than 90 trials. What is more significant 
about Fig. 1 is that the data points which had more than 90 
trials required a total of 5065 trials, whereas those requiring 
less than 90 trials required only a total of 3988 trials. This 
means that 13% of the data points required approximately 56% of 
the effort. Similarly, the upper 10% of the data points showed 
1586 errors out of a total of 2575 error. This means that 
approximately 61% of all the errors occurred in 10% of the data 
point. These two observations indicate that there should be 
some upper value of E and/or T which would signify the need 
for branching. That is, some point that, if reached, would 
indicate that the subject is having an exceptional amount of 
difficulty and should be branched. To develop this furt; 3r 
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requires returning to Eqs . (5) and (8). The following shows 

that these equations are true probability mass functions. 






l Pd-p ) 1 

i=0 



l-d-p) 



= 1 



Jlim 

t-*-°° 



P [T< t ] = 



p+ l p(l-g) (1-c) 1 
i=l 



p+p (1-g) (1-c) l (1-c) 1 ' 
i=l 



p+p (1-g) (l-c)| 



= pd+iiiaL ilzsl) 



, c+l-c-g (1-c) . 
p ' c ' 



= piiigt. 1 .-. 0 .)) 



■ e<±> * 1 

Based on this development it follows that 



P [E>k ] =1-1 pd-p) 1 

i=0 

_ , p[l-(l-p) k+1 ] 

l-(l-p) 



= 1 - 1 + (1-p) 



k + 1 



P [E>k ] = { l\ C) ] 

1-g (1-c) 



k+1 



1 



(13) 
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and 



P [T>t ] = 



1-p- l p(l-g) (l-c) 1 
i=l 



= (l-p)-p(l-g) (1-c) I (1-c) 

i=l 



i-1 



= (1-p) -p (1-g) (1-c) 



[l-(l-c) ] 



_ (1-g) (1-c) (1-g) (1-c) . t, 

~ 1-gTra l~"g”("l-cl [1(lc) J 






(14) 



With Eqs. (13) and (14) in hand, it simply becomes a matter 
of calculations. The calculations are similar to those done 
for the first criterion. To get the unconditional probability 
that E or T is greater than some number requires the following 
basic type equations, involving two independent random variables 
(g and c) . 



P [E>k] = 



and 

P [T>t ] 



H P [E>k | g=x, c=y] P[g=x) P[c=y] 
gc 

k+1 

^ t' l-xri ^ yf ] P[ 9 =x] P[c= y ] 



II P [L>t |g=x,c=y] P [g=x] P[c=y] 
gc 

II - i - Ifa-J’ p[g=xl Ptc=yl 

g c xv 



(15) 



(16) 



Tables IV and V represent the results for E and T respectively. 
Also listed in Tables IV and V are the values obtained om the 
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lower abscissa's of Figs. 2 and 1, respectively. The lower 
abscissa's represent the cumulative percentages, that is, the 
percentage of data points with values less than or equal to 
the value indicated on the first abscissa. For the sake of 
comparison, one minus the values in Figs. 2 and 1 are used 
in Tables IV and V. This value then represents the percentage 
of data points in which E and T were greater than the associated 
criterion number. 



TABLE IV 

ESTIMATED PROBABILITY THAT THE NUMBER OF ERRORS 
WILL EXCEED A GIVEN CRITERION k [P(E>k)] 



Criterion 


Estimate 


Observed 


k 


P (E>k) 


Figure 2 


5 


.29 


.36 


10 


.17 


.27 


15 


.12 


. .20 


20 


.08 


.14 


25 


.05 


.10 



(Also associated with this table are the observed 
percentages of data points which had more than k 
errors [from Fig. 2]). 



It should be noted that the observed values in Tables 
IV' and V are consistently higher than the expected values. 
Some thought indicated that the assumption of independence, 
required for the development of Eqs . (15) and (16) may n t 
have held up. A Pearson "r" test was run with g=X and t Y. 
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TABLE V 



ESTIMATED PROBABILITY THAT THE NUMBER OF TRIALS 
WILL EXCEED A GIVEN CRITERION t [P(T>t)] 



Criterion 

t 


Estimate 
P (T>t ) 


Observed 
Figure 1 


60 


.13 


.20 


70 


.10 


.18 


80 


00 

o 

• 


.16 


90 


.07 


.13 


100 


.06 


.12 



(Also associated with this table are the observed 
percentages of data points which had more than t 
trials [from Fig. 1]). 

The result was r=-.2774 with n=208. This value is significant 
at greater than the P=.01 level. With a knowledge of this fact an 
examination of Eqs . (15) and (16) indicates why the estimated 

values were lower than the observed values. It is felt however 

X A> 

that the correlation between g and c is an unavoidable artifact 
of the estimators. In explanation of this last statement, 
observe that Eqs. (11) and (12) have an upper and lower bound 
based on the value of T. These bounds are most restrictive at 
the lower values of T where c is larger than g has its most 
restrictive upper bound. Since 36% of the population had a T 
value of ten or less (Figv 1) , it is relatively easy to accept 

• A A 

a higher c and lower g (negative r) bias in the estimators. 

The branching criterion must therefore be established wit' an 
awareness of this artifact. 
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One case of branching was observed in the raw data. 

At that time the branching criterion required three consecu- 
tive sessions with a percent of accuracy less than 80%. It 
took 184 trials before branching was performed. Had a 
criterion of 25 errors been used, the subject would have 
branched after the 57th trial, which would obviously have 
been an advantageous move. Since there is no apparent way to 
measure the benefit of branching over just forcing the subject 
through the step, a comparison similar to that done for the 
first criterion cannot be made. It is felt however that the 
confidence on using the number of errors or the number of 
trials as branching criterion will lie somewhere between the 
estimated and the observed values (Tables IV and IV) . 
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IV. CONCLUSION 



In general, any progtam that has the form of those used 
in this study could be analyzed in a similar manner. Once 
the program is ready for testing, a representative sample 
should be run through with an arbitrarily high criterion, say 
20 consecutive correct responses with no branching. (Branching 
changes the basic probabilities (g and c) and confounds the 
data.) Analysis of the resulting data would then indicate 
the criteria to be used when the program is given general use. 
For this particular program the following criteria seem adequate 

Pass Criterion = 16 consecutive correct responses 

(88-92% confidence) 

Branch Criterion = 25 total errors (90-95% confidence) 
or Branch Criterion = 100 total trials (88-94% confidence) 

It is recommended that anyone using this procedure attempt 
to increase the number of data points at least twofold. This 
increased number of data points would smooth out the frequency 
distribution and possibly point more exactly to how c and g are 
distributed. Having a smoother frequency distribution will 
also fill in blanks in the joint probability matrix used in 
calculating P (E>k) and P (T>t) . With more cells filled in, it 
may be possible to use the actual joint density P(g=X, c=Y) 
rather than the product of the marginal densities P (g=X) *P (c=Y) . 
This would bring the estimated and observed values in Tables 
IV and V closer together because it would remove the requ re- 
ment for an assumption of independence which appears to 
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violated due to the model. Knowledge of 
before the data is taken will also allow 
of the data gathering. 



this procedure 
for better control 
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APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 



Subject 


A 1 


A 2 


A 3 


A 4 


A 5 


A 6 


A 7 


B 1 


B 2 


B 3 


B 4 


B 5 




12 


71 


47 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


4 


42 


18 


0 


0 


0 


0 


0 


0 


0 


0 


0 


2 


7 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 




7 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


























4 
























- 


5 


0 


0 


9 


0 


0 


0 


0 


0 


0 


0 


0 


0 




0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


6 


0 


2 


1 


0 


13 


0 


0 


0 


0 


0 


0 


0 




0 


2 


1 


0 


1 


0 


0 


0 


0 


0 


0 


0 


7 


























8 


























9 


4 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 




4 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


10 


0 


1 


1 


8 


8 


11 


76 


108 


219 


44 


99 


62 




0 


1 


1 


1 


1 


1 


13 


15 


41 


5 


25 


17 
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APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 

Subject C] _ C 2 C 3 C 4 C 5 Dl D 2 D 3 D 4 D 5 E] _ E 2 



1 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 0 0 0 
0 0 0 0 



2 



0 0 39 0 14 10 

0 0 12 0 2 2 



0 0 
0 0 



0 24 0 0 

0 5 0 0 



3 



37 0 0 

14 0 0 



8 0 0 1 

3 0 0 1 



4 



5 3 22 0 0 9 

3 1 8 0 0 5 



0 0 15 25 0 0 

0 0 4 13 0 0 



5 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



3 39 2 4 0 0 

15 110 0 



6 



39 

3 



2 0 12 0 0 0 

2 0 7 0 0 0 



0 22 0 0 0 

0 3 0 0 0 



7 



8 



41 0000 14 3 41 0600 

7000011 18 0500 

6 0 43 30 0 26 85 15 • 221 0 16 2 

205406 14 2 45 031 
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APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 

Subject E 3 E 4 E 5 F x P 2 F 3 Gj _ G 2 G 3 I, 



1 



0 0 0 
0 0 0 



0 0 0 109 0 

0 0 0 50 0 



0 

0 



54 

4 



2 



0 0 0 
0 0 0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 0 
0 0 



3 



0 0 9 11 

0 0 2 2 



0 9 0 69 8 

0 4 0 20 1 



0 8 
0 1 



4 



35 0 27 0 

5 0 5 0 



7 8 • 0 10 23 

110 6 6 



2 23 

1 7 



5 



0 0 0 0 17 3 

0 0 0 0 3 1 



3 0 3 2 

10 11 



0 

0 



6 



10 0 0 2 4 

3 0 0 1 1 



0 10 35 0 12 

0 4 6 0 1 



0 0 1 12 6 
0 0 13 2 



0 0 0 
0 0 0 



9 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 0 55 47 16 0 

0 0 5 5 2 0 



10 



2 57 0 

2 13 0 



5 43 37 

18 5 



0 310 129 16 

0 79 30 3 



/ 
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APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 

Subject A 1 A 2 A 3 A 4 A g A & A ? B 1 B 2 B 3 B 4 B 5 



0110000000 15 0 

011000000010 



0 3 0 0 91 48 178 0 3 3 0 0 

0200 34 9 67 01100 



0 32 138 277 12 18 0 21 59 8 13 5 

0 8 85 123 2 2 0 3 15 1 5 1 



14 



010000000000 

010000000000 



15 



16 



0 14 0 1 96 9 2 

0 3 0 1 19 2 1 



0 38 106 0 0 

0 2 19 0 0 



17 



1 1 
1 1 



0 0 0 2 0 0 0 
0 0 0 1 0 0 0 



0 

0 



0 

0 



0 

0 



18 



44 1 0 

10 1 0 



000000000 

000000000 



19 • 



0 0 0 
0 0 0 



000010001 
000010 0 01 



37 



APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 

Subject Cj C 2 C 3 C 4 C 5 ^ D., D, D,. E 2 



11 


10 


0 


0 


0 


0 




1 


0 


0 


0 


0 


12 


20 


36 


269 


6 


212 


10 


10 


99 


1 


104 


13 


170 


0 


28 


16 


0 


93 


0 


4 


1 


0 


14 


0 


0 


0 


0 


0 




0 


0 


0 


0 


0 


15 


- 


- 


- 


- 


- 


16 


0 


7 


0 


0 


0 




0 


3 


0 


0 


0 


17 


0 


0 


0 


9 


10 




0 


0 


0 


1 


2 


18 


0 


0 


0 


0 


0 




0 


0 


0 


0 


0 


19 


0 


0 


0 


0 


0 




0 


0 


0 


0 


0 



0 


0 


0 


0 


19 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


0 


0 


334 


0 


262 


55 


0 


0 


0 


119 


0 


57 


17 


233 


53 


37 


0 


200 


133 


54 


78 


24 


9 


0 


39 


40 


14 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


27 


21 


6 


28 


0 


43 


78 


6 


3 


5 


4 


0 


9 


16 


11 


0 


0 


0 


4 


0 


52 


3 


0 


0 


0 


1 


0 


24 


0 


15 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


23 


4 


0 


0 


0 


0 


0 


3 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 
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APPENDIX A 



Individual Data Matrix (T and E) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 



subject e 3 e 4 e 5 p x f 2 f 3 g x g 2 g 3 h x 



11 



0 0 0 0 17 

0 0 0 0 1 



0 21 0 8 0 0 
0 2 0 1 0 0 



12 



16 101 
4 15 



12 

3 



0 142 

0 32 



7 

1 



9 

3 



13 

3 



4 

1 



0 

0 



0 

0 



13 



12 80 49 47 178 104 23 64 334 0 88 

2 18 11 17 44 19 2 24 112 0 17 



14 



0 

0 



0 

0 



0 

0 



0 

0 



9 

1 



0 

0 



1 

1 



86 

14 



6 

1 



0 

0 



0 

0 



15 



70 

22 



6 

3 



9 

3 



37 

4 



0 

0 



9 

3 



13 184 

4 86 



0 

0 



82 

32 



0 

0 



16 



0 

0 



0 

0 



0 

0 



0 

0 



22 

2 



0 

0 



0 

0 



50 

23 



0 

0 



0 

0 



0 

0 



17 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



36 

6 



41 

13 



0 

0 



6 

1 



0 

0 



18 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



75 

15 



36 

3 



0 

0 



0 

0 



0 

0 



19 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 



0 

0 
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APPENDIX B 



Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 

A A 

Top Entry Equals g, Bottom Entry Equals c 
(S = Tested with No Errors, S' = Not Tested) 

Subject A 2 A 3 A 4 A g A g A ? B 1 B 2 B 3 B 5 



.750 .570 .644 
.067 .014 .020 



SSSSSSSSS 



,222 

,125 



S ' S ' S ' S ' S ' S ' S ' 



4 

5 



S ' S ' S ' 

.770 

1 .083 S 



S' S' S' S' S' S' S' s' 



.500 .667 
.333 .500 



,963 

,038 



■ I C » 



S' S' s 1 



8 S' S' S' S' S' S' S' S' S' S' 



333 .500 _______ 

sss s ssssss 



,200 



,333 



10 



.667 .667 .941 .941 .956 .843 .871 .818 .907 .760 .745 
.500 .500 .063 .063 .045 .012 .009 .004 .019 .010 .015 
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APPENDIX B 



Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 

✓S A 

Top Entry Equals g, Bottom Entry Equals c 
(S = Tested with No Errors, S' = Not Tested) 



Subject 


C 1 C 2 


C 3 


C 4 


C 5 


D 1 


°2 


°3 


°4 


°5 


E 1 


E 2 


i 


.927 .911 


.610 


s 


s 


s 


s 


s 


S 


S • 


s 


s 




.025 .023 


.008 




















2 


s s 


.723 


s 


.909 


.875 


s 


s 


s 


.832 


s 


s 






.024 




.048 


.067 








.035 






3 


s' s' 


s' 


s' 


s' 


.656 


s 


s 


.743 


S 


s 


.667 












.025 






.094 






.500 


4 


.609 .857 


.689 


s 


s 


.576 


s 


s 


.797 


.534 


s 


S 




.150 .167 


.040 






.093 






.053 


.037 






5 


S S 


S 


s 


s 


S 


.857 


.845 


.800 


.889 


s 


S 














.167 


.021 


.250 


.125 






6 


.943 .500 


S 


.524 


s 


S 


S 


S 


.901 


S 


s 


s 




.019 .333 




.073 










.034 








7 


S' S' 


S' 


S' 


S' 


S' 


S' 


S' 


S' 


S ' 


S' 


S ' 


8 


S' S' 


S' 


S' 


S' 


s ' 


S' 


S' 


S' 


S ' 


S' 


s' 


9 


.854 s 


s 


s 


s 


.965 


.857 


.593 


S 


.390 


s 


s 




.021 








.036 


.125 


.023 




.139 






10 


.BOO s 


.905 


.896 


s 


.809 


.848 


.915 


.802 


S 


. 866 


.800 




.111 


.019 


.027 




.033 


.011 


.044 


.004 




.047 


.250 
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Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals g, Bottom Entry Equals c 





(S 


= Tested 


with 


No Errors, 


S ' 


= Not 


Tested) 




Subject 


E 4 


E 5 


F 1 


F 2 


F 3 


G 1 


G 2 


G 3 


H 1 


*1 


1 


s 


s 


s 


s 


s 


S 


.554 


S 


S 


.941 


s' 
















.009 






.015 




2 


s 


s 


s 


s 


s 


S 


S 


s 


S 
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APPENDIX B 



Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals g, Bottom Entry Equals c 
(S = Tested with No Errors, S' = Not Tested) 
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APPENDIX B 



Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals g. Bottom Entry Equals c 
(S = Tested with No Errors, S' = Not Tested) 
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APPENDIX B 



Individual Data Matrix (g and c) 

Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals g. Bottom Entry Equals c 
(S = Tested with No Errors, S' = Not Tested) 
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91 to 334. All occurred once except for T=334 which 
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Figure 4. Frequency Distribution of The Estimated Presolution 
Probability of a Correct Response or Guess (g) 
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Figure 5. Frequency Distribution of The Mean Values (BY SUBJECT) 
of The Estimated Probability of Going Into Solution (c 
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Figure 6. Frequency Distribution of The Mean Values (BY STEP) 

of The Estimated Probability of Going Into Solution (c 
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of The Estimated Presolution Probability of a Correct 
Response or Guess (g) . 
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